Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus generates first image data based on a first white balance correction value corresponding to a first light source, and second image data based on second white balance correction value corresponding to a second light source, selects one white evaluation region for determining a combination ratio from a plurality of white evaluation regions, compares a color evaluation value of the image data with the selected white evaluation region under the second light source, determines a combination ratio of the first image data and the second image data based on a result of the comparison, and combines the first image data and the second image data according to the determined combination ratio.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a white balance control technology forimage data.

2. Description of the Related Art

There has conventionally been known a method for adjusting white balance(WB) based on a signal output from an image sensor. As an example,automatic white balance (AWB) processing is known. The automatic whitebalance control is for automatically detecting a portion likely to bewhite from a captured image data, calculating a white balancecoefficient from an average value among color components of the entirescreen, and then applying the calculated white balance coefficient tothe entire screen.

Such automatic white balance processing has the following problems. In ascene where a plurality of different light sources is present, thecalculated white balance coefficient is applied to the entire screen. Itis, therefore, difficult to carry out white balance control forachieving appropriate colors for the respective light sources. Forexample, it may be supposed that while a main object is illuminated by amercury lamp in a nightscape scene, a light source for a group ofbuildings in the background is a fluorescent lamp. In such a scene, whenthe white balance of the main object under the mercury lamp light sourceis set appropriate, a florescent lamp light source portion of thebackground becomes reddish. Similarly, when the white balance of thebackground is set appropriate, the white balance of the mercury lamplight source portion of the main object shifts from an appropriatestate. Further, when white balance control is carried out to setintermediate white balance appropriately between the light sources, bothlight source portions are colored.

Further, in many cases, a color evaluation value of each block under themercury lamp light source cannot be detected within a white detectionrange under a normal light source. Even in a single light source sceneof the mercury lamp, it is difficult to acquire appropriate whitebalance.

Japanese Patent Application Laid-Open No. 2000-212641 discusses thefollowing technology. That is, according to the technology discussed inJapanese Patent Application Laid-Open No. 2000-212641, when the numberof blocks determined to be white within a first white detection range issmaller than a fixed number, whether photographing is under a low colortemperature light source is determined. When it is determined that thephotographing is under the low color temperature light source, a whitedetection range of the low color temperature side is widened to be setas a second white detection range. Then, when the number of blocksdetermined to be white within the second white detection range is equalto or larger than the fixed number, a white balance correctioncoefficient is calculated based on the blocks determined to be white.

However, according to the technology discussed in Japanese PatentApplication Laid-Open No. 2000-212641, when it is determined that thephotographing is under the low color temperature light source, the whitedetection range is widened, and the white balance correction coefficientis calculated based on the new blocks determined to be white. Thus, thetechnology cannot deal with a case where a plurality of light sources ispresent.

SUMMARY OF THE INVENTION

The present invention is directed to an image processing apparatus andan image processing method for generating image data of an appropriatecolor even when a plurality of light sources are mixed.

According to an aspect of the present invention, an image processingapparatus includes a correction value determination unit configured todetermine a first white balance correction value corresponding to afirst light source in captured image data and a second white balancecorrection value corresponding to a second light source in the imagedata, a generation unit configured to generate first image data bycorrecting the image data based on the first white balance correctionvalue, and second image data by correcting the image data based on thesecond white balance correction value, a color evaluation valuecalculation unit configured to calculate a color evaluation value of theimage data, a selection unit configured to select one white evaluationregion for determining a combination ratio from a plurality of whiteevaluation regions, a combination ratio determination unit configured tocompare the color evaluation value of the image data with the whiteevaluation region under the second light source selected by theselection unit, and to determine a combination ratio of the first imagedata and the second image data based on a result of the comparison, anda combining unit configured to combine the first image data and thesecond image data according to the combination ratio determined by thecombination ratio determination unit.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram illustrating a configuration example of animage processing apparatus according to an exemplary embodiment of thepresent invention.

FIG. 2 is a flowchart illustrating a method for calculating a firstwhite balance correction value (first WB correction value).

FIG. 3 is a graph used for white detection.

FIG. 4 is a flowchart illustrating combining processing of image data byan image processing circuit.

FIG. 5 illustrates a relationship between color evaluation values usedfor combination ratio calculation.

FIG. 6 illustrates a relationship between color evaluation values usedfor combination ratio calculation when a face is detected.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 is a block diagram illustrating a configuration example of animage processing apparatus according to an exemplary embodiment of thepresent invention. In FIG. 1, the image processing apparatus 100includes a photographic lens 10, a shutter 12 having a diaphragmfunction, an image sensor 14 that converts an optical image into anelectric signal, an analog/digital (A/D) converter 16 that converts ananalog signal output from the image sensor 14 into a digital signal, anda timing generation circuit 18 that supplies a clock signal or a controlsignal to the image sensor 14, the A/D converter 16, and adigital/analog (D/A) converter 26 and that is controlled by a memorycontrol circuit 22 and a system control circuit 50.

An image processing circuit 20 executes predetermined pixelinterpolation processing or color conversion processing for data fromthe A/D converter 16 or data from the memory control circuit 22. Theimage processing circuit 20 carries out autofocus (AF) processing of athrough-the-lens (TTL) method, automatic exposure (AE) processing, andflash pre-emission (EF) processing where predetermined calculationprocessing is executed by using captured image data, and the systemcontrol circuit 50 executes control for an exposure control unit 40 anda focusing control unit 42 based on the acquired calculation result.Further, the image processing circuit 20 carries out predeterminedcalculation processing by using the captured image data, and automaticwhite balance processing of a TTL method based on the acquiredcalculation result.

The memory control circuit 22 controls the A/D converter 16, the timinggeneration circuit 18, the image processing circuit 20, an image displaymemory 24, the D/A converter 26, a memory 30, and acompression/decompression circuit 32. Data of the A/D converter 16 iswritten in the image display memory 24 or the memory 30 via the imageprocessing circuit 20 or the memory control circuit 22, or directly bythe memory control circuit 22.

Image data for displaying, which has been written in the image displayunit 24, is passed through the D/A converter 26 to be displayed by animage display unit 28 including a thin-film transistor liquid crystaldisplay (TFT LCD). Sequentially displaying the captured image data byusing the image display unit 28 can realize an electronic viewfinderfunction. The image display unit 28 can arbitrarily turn ON/OFFdisplaying according to an instruction from the system control circuit50. When the displaying is turned OFF, power consumption of the imageprocessing apparatus 100 can be greatly reduced.

The memory 30, which stores captured still image data or moving imagedata, has a storage capacity sufficient for storing a predeterminednumber of pieces of still image data or moving image data of apredetermine period of time. Accordingly, even in the case of continuousphotographing for continuously capturing a plurality of still image dataor a panoramic photographing, a great volume of images can be written inthe memory 30 at a high speed. Further, the memory 30 can be used as awork area for the system control circuit 50.

The compression/decompression circuit 32, which compresses/decompressesthe image data by adaptive discrete cosine conversion (ADCT), reads theimage stored in the memory 30 to compress or decompress it, and writesthe processed data in the memory 30.

The exposure control unit 40, which controls the shutter 12 having thediaphragm function, has a flash light adjustment function by associationwith a flash unit 48. The focusing control unit 42 controls focusing ofthe photographic lens 10. A zoom control unit 44 controls zooming of thephotographic lens 10. A barrier control unit 46 controls an operation ofa projection unit 102.

The flash unit 48 has an AF auxiliary light projection function and aflash light adjustment function. The number of flash units can be one,or a plurality of flash units such as a xenon (Xe) tube for flash lightemission and a light emitting diode (LED) for light emission of veryshort periods of time can be prepared. The exposure control unit 40 andthe focusing control unit 42 are controlled by using the TTL method. Thesystem control circuit 50 controls the exposure control unit 40 and thefocusing control unit 42 based on the result of calculating the capturedimage data by the image processing circuit 20.

The system control circuit 50 controls the entire image processingapparatus 100. A memory 52 stores a constant, a variable, and a programfor operating the system control circuit 50. The memory 52 furtherstores a program diagram used for the AE. The program diagram is a tabledefining a relationship between an exposure value and a control value ofan aperture value and a shutter speed.

A display unit 54 of a liquid crystal display apparatus 28 displays anoperation state or a message by using a character, an image, or a voicein response to execution of the program at the system control circuit50. A single or a plurality of display units 54 is disposed in an easilyviewed place near an operation unit of the image processing apparatus100, and configured by combining, for example, an LCD, an LED, and asound generation element.

Certain functions of the display unit 54 are installed in an opticalviewfinder 104. Among display contents of the display unit 54, those tobe displayed on the LCD include: a single-shot/continuous-shot display,a self-timer display, a compression rate display, a recording pixelnumber display, a recording frame number display, a remainingphotographable number display, a shutter speed display, an aperturevalue display, an exposure compensation display, a flash display, ared-eye reduction display, a macro-shot display, a buzzer settingdisplay, a clock battery remaining amount display, a battery remainingamount display, an error display, an information display based on aplurality of digits of numerals, an attached/detached state display ofrecording media 200 and 210, a communication interface (I/F) operationdisplay, and a date/time display.

Among the display contents of the display unit 54, those to be displayedin the optical viewfinder 104 include: an in-focus display, acamera-shake warning display, a flash charge display, a shutter speeddisplay, an aperture value display, and an exposure compensationdisplay.

For an electrically erasable/recordable nonvolatile memory 56, forexample, an electrically erasable programmable read-only memory (EEPROM)is used. Operation units 60, 62, 64, and 66 for inputting variousoperation instructions are configured each by a switch, a dial, a touchpanel, a pointing device based on visual line detection or a voicerecognition device, or combining a plurality of units.

The operation units will specifically be described. The mode dial switch60 can switch and set function modes including a power-off mode, anautomatic photographing mode, a photographing mode, a panoramicphotographing mode, a reproduction mode, a multiscreenreproduction/erasure mode, a personal computer (PC) connection mode. Theshutter switch 62 (SW1) is turned ON during an operation of a shutterbutton (not illustrated) to instruct an operation start of AFprocessing, AE processing, AWB processing, or EF processing.

The shutter switch 64 (SW2) is turned ON when the operation of theshutter button (not illustrated) is completed to instruct an operationstart of a series of processes, namely, exposure processing for writingimage data of a signal read from the image sensor 14 in the memory 30via the A/C converter 16 and the memory control circuit 22, developmentprocessing using calculation at the image processing circuit 20 or thememory control circuit 22, and recording processing for reading theimage data from the memory 30, compressing the image data by thecompression/decompression circuit 32, and writing the image data in therecording medium 200 or 210.

The operation unit 66 includes various buttons and a touch panel:specifically, a menu button, a set button, a macrobutton, an imagedisplay ON/OFF button, a multiscreen reproduction page break button, aflash setting button, a single-shoot/continuous-shoot/self-timerswitching button, a menu movement + (plus) button, a menu movement −(minus) button, a reproduced image movement + (plus) button, areproduced image movement − (minus) button, a captured image qualityselection button, an exposure compensation button, and a date/timesetting button.

A power source control unit 80 includes a battery detection circuit, adirect current (DC)-DC converter, and a switch circuit for switching ablock to be energized. The power source control unit 80 detects presenceof a loaded battery, a type of the battery, or a remaining batteryamount, controls the DC-DC converter based on the detection result andan instruction from the system control circuit 50, and supplies anecessary voltage to portions including a recording medium for anecessary period.

The image processing apparatus 100 further includes connectors 82 and84, a power source unit 86 including a primary battery such as analkaline battery or a lithium battery, a secondary battery such as aNiCd battery, a NiMH battery, or a Li battery, and an alternativecurrent (AC) adaptor, interfaces 90 and 94 with a recording medium suchas a memory card or a hard disk, connectors 92 and 96 for connectionwith the recording medium such as a memory card or a hard disk, and arecording medium attachment/detachment detection unit that detectsloading of the recording medium 200 or 210 in the connector 92 or 96.

In the exemplary embodiment, the image processing apparatus 100 includesthe two interfaces and the two connectors. However, the numbers ofinterfaces and connectors for attaching the recording medium can besingular or plural. For the interface and the connector, those compliantwith a standard of a Personal Computer memory Card InternationalAssociation PCMCIA) card or a CompactFlash (CF, registered trademark)card can be used.

When for the interfaces 90 and 94 and the connectors 92 and 96, thosecompliant with the standard of the PCMCIA card or the CF card can beused, by connecting various communication cards such as a local areanetwork (LAN) card or a modem card, a universal serial bus (USB) card,an Institute of Electrical and Electronics Engineers (IEEE) 1394 card, aP1284 card, a small computer system interface (SCSI) card, and apersonal handyphone system communication card, image data and managementinformation accessory to the image data can be transferred with othercomputers or peripheral devices such as a printer. The protection unit102 is a barrier for covering an imaging unit including the lens 10 ofthe image processing apparatus 100 to prevent staining or damaging ofthe imaging unit.

Photographing can be carried out by using only the optical viewfinder104 without using the electronic viewfinder function of the imagedisplay unit 28. In the optical viewfinder 104, some functions of thedisplay unit 54, such as the in-focus display, the hand-shake warningdisplay, the flash charge display, the shutter speed display, theaperture value display, and the exposure compensation display, areinstalled.

A communication unit 110 has various communication functions such asRS232C or USB, IEEE1394, P1284, SCSI, a modem, a LAN, and wirelesscommunication. A connector 112, or an antenna in the case of wirelesscommunication, connects the image processing apparatus 100 to otherdevices by the communication unit 110. The recording medium 200 is amemory card or a hard disk. The recording medium 200 includes arecording unit 202 including a semiconductor memory or a magnetic disk,an interface 204 with the image processing apparatus 100, and aconnector 206 for connection with the image processing apparatus 100.The recording medium 210 is a memory card or a hard disk. The recordingmedium 210 includes a recording unit 212 including a semiconductormemory or a magnetic disk, an interface 214 with the image processingapparatus 100, and a connector 216 for connection with the imageprocessing apparatus 100.

Next, a method for calculating a WB correction value by the imageprocessing circuit 20 according to the exemplary embodiment will bedescribed. This processing is for calculating a WB correction valuebased on WB control using a blackbody locus. First, referring to FIG. 2,a method for calculating a first white balance correction value (firstWB correction value) will be described. The first WB correction value iscalculated by normal white balance control for a first light source.

In step S201, the image processing circuit 20 reads the image datastored in the memory 30, and divides the image data into predetermined mblocks. In step S202, the image processing circuit 20 adds and averagespixel values for respective colors for blocks (1 to m) to calculateaverage color values (R[i], G[i], B[i]), and calculates a colorevaluation value (Cx[i], Cy[i]) by using the following expressions:Cx[i]=(R[i]−B[i])/Y[i]×1024Cy[i]=(R[i]+B[i]−2G[i])/Y[i]×1024where Y[i]=(R[i]+2G[i]+B[i])/4.

In step S203, the image processing circuit 20 determines whether thecolor evaluation value (Cx[i], Cy[i]) of an i-th block is within apreset white detection range 301 illustrated in FIG. 3. The whitedetection range 301 is set by first photographing a white object underdifferent light sources and then plotting calculated color evaluationvalues. The white detection range can be set individually according to aphotographing mode. In FIG. 3, a negative direction of the x coordinate(Cx) represents a color evaluation value when white object isphotographed with a high color temperature light source, while apositive direction thereof represents a color evaluation value whenwhite object is photographed with a low color temperature light source.The y coordinate (Cy) indicates a degree of a green component in a lightsource, and the green component is larger toward a negative direction ofthe y coordinate.

When the color evaluation value (Cx[i], Cy[i]) of the i-th block iswithin the preset white detection range 301 (YES in step S203), theprocessing proceeds to step S204. On the other hand, when the colorevaluation value (Cx[i], Cy[i]) of the i-th block is not within thepreset white detection range 301 (NO in step S203), the processing skipsstep S204 to proceed to step S205.

In step S204, the image processing circuit 20 determines that the i-thblock is white to integrate the average color values (R[i], G[i], B[i])of the block. The processing of steps S203 and S204 can be representedby the following expressions:

${SumR} = {\sum\limits_{i = 0}^{m}{{{Sw}\lbrack i\rbrack} \times {R\lbrack i\rbrack}}}$${SumG} = {\sum\limits_{i = 0}^{m}{{{Sw}\lbrack i\rbrack} \times {G\lbrack i\rbrack}}}$${SumB} = {\sum\limits_{i = 0}^{m}{{{Sw}\lbrack i\rbrack} \times {B\lbrack i\rbrack}}}$

In the expressions, when the color evaluation value (Cx[i], Cy[i]) iswithin the white detection range 301, Sw[i] is set to 1. When the colorevaluation value (Cx[i], Cy[i]) is not within the white detection range301, Sw[i] is set to 0. Accordingly, in steps S203 and S204, whether tointegrate the average color values (R[i], G[i], and B[i]) issubstantially determined.

In step S205, the image processing circuit 20 determines whether theprocessing has been completed for all the blocks. When not completed (NOin step S205), the processing returns to step S202 to be repeated. Onthe other hand, when completed (YES in step S205), the processingproceeds to step S206.

In step S206, the image processing circuit 20 calculates a first WBcorrection value (WBCo1_R1, WBCo1_G1, WBCo1_B1) from integrated values(SumR1, SumG1, SumB1) of the acquired average color values by using thefollowing expressions:WBCo1_(—) R1=SumY1×1024/SumR1WBCo1_(—) G1=SumY1×1024/SumG1WBCo1_(—) B1=SumY1×1024/SumB1where Sum_Y1=(SumR1+2×SumG1+SumB1)/4.

Next, a method for determining a second white balance correction value(second WB correction value) for a second light source will bedescribed. The second WB correction value (WBCo2_R, WBCo2_G, WBCo2_B) isdetermined by using predetermined values for the respective identifiedlight sources. For these values, values calculated beforehand byphotographing white objects under the respective light sources are used.For example, in a scene where a mercury lamp is used for the lightsource, a WB correction value determined beforehand for the mercury lampis used as the second WB correction value. In this case, when aplurality of types of light sources are present, the second WBcorrection value can be varied from one type to another. For example,when two types of mercury lamps are present, the WB correction value canbe varied between the mercury lamps.

Next, referring to a flowchart of FIG. 4, combining processing of theimaged data of the image processing circuit 20 will be described. Instep S401, the image processing circuit 20 determines whether to executeassociation with the special light source. In this case, the imageprocessing circuit 20 can estimate a light source based on the imagedata stored in the memory 30, and determine whether to executeassociation with the special light source according to the estimationresult, or determine whether to execute association with the speciallight source according to a result designated beforehand by the user.When association with the special light source is executed (YES in stepS401), the processing proceeds to step S402. On the other hand, whenassociation with the special light source is not executed (NO in stepS401), the processing proceeds to step S410.

In step S402, the image processing circuit 20 calculates a first WBcorrection value WBCo1 by the method described above referring to FIG.2. In step S403, the image processing circuit 20 calculates a second WBcorrection value WBCo2 by the method. In step S404, the image processingcircuit 20 develops development image data Yuv1 from the image datastored in the memory 30 by using the first WB correction value WBCo1. Instep S405, the image processing circuit 20 develops development imagedata Yuv2 from the image data stored in the memory 30 by using thesecond WB correction value WBCo2. The development image data Yuv1 is anexample of first image data, and the development image data Yuv2 is anexample of second image data.

In step S410, the image processing circuit 20 calculates a first WBcorrection value WBCo1 by the same method as that in step S402. In stepS411, the image processing circuit 20 develops development image dataYuv1 from the image data stored in the memory 30. In other words, theimage processing circuit 20 executes normal white balance control.

In step S406, the image processing circuit 20 divides each of the imagedata stored in the memory 30, the development image data Yu1, and thedevelopment image data Yu2 into n blocks.

In step S407, the image processing circuit 20 adds and averages pixelvalues for the respective colors for each block of the image data storedin the memory 30 to calculate average color values (R[i], G[i], B[i]).Then, as in the case of step S202, the image processing circuit 20calculates a color evaluation value (Cx[i], Cy[i]). The color evaluationvalue (Cx[i], Cy[i]) calculated in step S202 can directly be used. Inthis case, when a saturated pixel is present, a pixel value of thesaturated pixel and a pixel value of a pixel of another colorcorresponding to the saturated pixel do not need to be included inaddition processing. For example, when a certain R pixel is a saturatedpixel, a pixel value of the R pixel, and a pixel value of a G pixel anda pixel value of a B pixel corresponding to the R pixel are not includedin the addition processing.

In step S408, the image processing circuit 20 sets an evaluation frameand an inner frame described below according to each light source, andcalculates a combination ratio of each block based on a differencebetween the second WB correction value and the color evaluation value ofeach block. For example, in a scene using a mercury lamp as a lightsource, as illustrated in FIG. 5, the image processing circuit 20 setsan evaluation frame 501 and an inner frame 502 for the mercury lamp byusing, as a center of gravity, a predetermined white evaluation valueunder the mercury lamp light source. Then, the image processing circuit20 calculates a combination ratio of each block based on a distancebetween the predetermined white evaluation value under the mercury lamplight source and the color evaluation value of each block. Theevaluation frame 501 is set based on a color evaluation value calculatedbeforehand by photographing a white object under the mercury lamp lightsource.

First, for a block where a difference between the color evaluation valueand the white evaluation value under the mercury lamp light source issmall and a color evaluation value is present in the inner frame 502illustrated in FIG. 5, a combination ratio α[i] of the block is setto 1. Then, for a block where a color evaluation value is present in aregion between the evaluation frame 501 and the inner frame 502illustrated in FIG. 5, the image processing circuit 20 calculates acombination ratio α[i] as follows. The image processing circuit 20linearly reduces a combination ratio from the inner frame 502 to theevaluation frame 501 to set the inner frame 502 to combination ratioα[i]=1 and the evaluation frame 501 to combination ratio α[i]=0, andcalculates a combination ratio α[i] of the block. For a block where acolor evaluation value is present outside the evaluation frame 501, theimage processing circuit 20 sets a combination ratio α[i] to 0.

In this case, pluralities of evaluation frames and inner frames arestored beforehand to be selectable according to an identified condition.For example, referring to FIG. 6, a case where a face is detected underthe mercury lamp light source as an identified condition will bedescribed. As indicated by a frame 603, blocks each of which includes aface under the mercury lamp light source tend to be distributed outsidethe range of the evaluation frame 501 illustrated in FIG. 5. In view ofsuch a new distribution, when a face is detected, the evaluation frameand the inner frame are respectively changed to a new evaluation frame601 and a new inner frame 602. As conditions for selecting theevaluation frame 601 and the inner frame 602, in addition to thepresence of the face, a condition concerning a special light sourcecorrection degree (e.g., user selects one of strong, middle, and weak asstrength of correction) designated by the user, or credibility of lightsource estimation (e.g., determined from a ratio of green regionincluded in image signal or coordinates of distribution of green regionin the case of mercury lamp) can simultaneously be used.

The evaluation frame and the inner frame can be formed into arbitraryshapes in place of square shapes. Further, for example, even in a scenethat uses a low color temperature light source such as a bulb lightsource, an evaluation frame and an inner frame suited to the lightsource are set, and a combination ratio can be calculated by processingsimilar to the combination ratio calculation method in the scene underthe mercury lamp light source.

In step S409, the image processing circuit 20 combines the firstdevelopment image data Yuv1 and the second development image data Yuv2by using the combination ratio α[i] of each block to generate combinedimage data Yuv3. To calculate a color evaluation value (Y3[i], u3[i],v3[i]) of the combined image data Yuv3, a color evaluation value (Y1[i],u1[i], v1[i]) of the development image data Yuv1 and a color evaluationvalue (Y2[i], u2[i], v2[i]) of the development image data Yuv2 are used.In other words, the image processing circuit 20 calculates the colorevaluation value (Y3[i], u3[i], v3[i]) of the combined image data Yuv3by the following expressions:Y3[i]=Y1[i]*(1−α[i])+Y2[i]*α[i]u3[i]=u1[i]*(1−α[i])+u2[i]*α[i]v3[i]=v1[i]*(1−α[i])+v2[i]*α[i]

To reduce color shifting generated on a boundary portion of the blocks,by further executing pixel interpolation processing in step S408, theimage processing circuit 20 can calculate a combination ratio α′[j] foreach pixel from the combination ratio α[i] of each block. For example,the image processing circuit 20 calculates the combination ratio α′[j]for each pixel from the combination ratio α[i] of each block by usingbilinear interpolation as pixel interpolation processing. In step S409,the image processing circuit 20 combines the development image data Yuv1and the development image data Yuv2 by using the combination ratio α′[j]for each pixel to generate combined image data Yuv3. To calculate acolor evaluation value (Y3[j], u3[j], v3[j]) of the combined image dataYuv3, the image processing circuit 20 uses a color evaluation value(Y1[j], u1[j], v1[j]) of the development image data Yuv1 and a colorevaluation value (Y2[j], u2[j], v2[j]) of the development image dataYuv2. In other words, the image processing circuit 20 calculates thecolor evaluation value (Y3[j], u3[j], v3[j]) of the combined image dataYuv3 by the following expressions:Y3[j]=Y1[j]*(1−α′[j])+Y2[j]*α′[i]u3[j]=u1[j]*(1−α′[j])+u2[j]*α′[j]v3[j]=v1[j]*(1−α′[j])+v2[j]*α′[i]

To reduce the amount of calculation at the image processing circuit 20,only the u component and the v component that are color components canbe combined. For example, the image processing circuit 20 calculates thecolor evaluation value (Y3[i], u3[i], v3[i]) of the combined image dataYuv3 by the following expressions:Y3[i]=Y1[i]u3[i]=u1[i]*(1−α[i])+u2[i]*α[i]v3[i]=v1[i]*(1−α[i])+v2[i]*α[i]

In the combining processing of the image data, the Yuv form is used forthe development image data. However, for example, when the RGB form isused as an image form, in place of the expressions used in step S409,the following expressions are used. That is, the image processingcircuit 20 calculates a color evaluation value (R1[i], G1[i], B1[i]) ofdevelopment image data RGB1 developed by using the first WB correctionvalue and a color evaluation value (R2[i], G2[i], B2[i]) of developmentimage data RGB2 developed by using the second WB correction value. Then,the image processing circuit 20 calculates a color evaluation value(R3[i], G3[i], B3[i]) of combined image data RGB3 by the followingexpressions:R3[i]=R1[i]*(1−α[i])+R2[i]*α[i]G3[i]=G1[i]*(1−α[i])+G2[i]*α[i]B3[i]=B1[i]*(1−α[i])+B2[i]*α[i]

In the combining processing of the image data, when two or more types oflight sources are present, two or more pieces of development image datacan be combined. For example, when flash light is emitted in the sceneusing the mercury lamp, the image processing circuit 20 determines, inaddition to the first WB correction value and the second WB correctionvalue, a third WB correction value as in the case of the method fordetermining the second WB correction value. Then, the image processingcircuit 20 develops three pieces of image data by using the three WBcorrection values to combine the image data.

Thus, according to the present exemplary embodiment, in the scene wherethe plurality of different light sources is present, image data forsetting both the main object and the background to appropriate colors isgenerated, and an image suitable for the user can be provided.

Next, another exemplary embodiment of the present invention will bedescribed. A processing procedure up to combining of image data in theabove-described exemplary embodiment is similar to that of the foregoingembodiment except for step S408.

In an example of a scene using the mercury lamp, in the above-describedexemplary embodiment, when a face is detected, the combination ratio iscalculated by using not the evaluation frame 501 and the inner frame 502illustrated in FIG. 5 but the evaluation frame 601 and the inner frame602 illustrated in FIG. 6. On the other hand, according to the presentexemplary embodiment, only when a combination ratio of a block includinga face region and a block near the block is calculated, the evaluationframe 601 and the inner frame 602 are used. In other cases, theevaluation frame 501 and the inner frame 502 are used. As a result, WBcorrection can be carried out more appropriately for a block unrelatedto a human object.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment (s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment (s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., non-transitory computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application No.2011-195402 filed Sep. 7, 2011, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a correction value determination unit configured to determine a first white balance correction value corresponding to a first light source in captured image data and a second white balance correction value corresponding to a second light source in the image data; a generation unit configured to generate first image data by correcting the image data based on the first white balance correction value, and second image data by correcting the image data based on the second white balance correction value; a color evaluation value calculation unit configured to calculate a color evaluation value of the image data; a selection unit configured to select one color evaluation region for determining a combination ratio from a plurality of color evaluation regions; a combination ratio determination unit configured to compare the color evaluation value of the image data with the color evaluation region under the second light source selected by the selection unit, and to determine a combination ratio of the first image data and the second image data based on a result of the comparison; and a combining unit configured to combine the first image data and the second image data according to the combination ratio determined by the combination ratio determination unit.
 2. The image processing apparatus according to claim 1, further comprising a face detection unit configured to detect a face from the image data, wherein the selection unit is configured to select a first color evaluation region when no face is detected by the face detection unit, and a second color evaluation region wider than the first color evaluation region when a face is detected by the face detection unit.
 3. The image processing apparatus according to claim 2, wherein the second color evaluation region includes more regions where skin color is present under the second light source than the first color evaluation region.
 4. The image processing apparatus according to claim 1, wherein the second light source includes a mercury lamp.
 5. The image processing apparatus according to claim 1, wherein the color evaluation value calculation unit is configured to calculate a color evaluation value of the image data for each block; and wherein the combination ratio determination unit is configured to determine the combination ratio for each block based on a difference between the color evaluation value of the image data for each block and the white evaluation value under the second light source.
 6. The image processing apparatus according to claim 5, wherein the combination ratio determination unit is configured to determine a combination ratio for each pixel from the combination ratio for each block by pixel interpolation processing.
 7. The image processing apparatus according to claim 1, wherein the correction value determination unit is configured to determine the first white balance correction value based on white balance control using a blackbody locus.
 8. The image processing apparatus according to claim 1, further comprising: an estimation unit configured to estimate a light source for the image data; and a determination unit configured to determine whether to execute combining processing by the combining unit according to a result of estimation of the light source by the estimation unit.
 9. The image processing apparatus according to claim 1, wherein the color evaluation calculation unit is configured not to use, when calculating the color evaluation value of the image data, a pixel value of a saturated pixel and a pixel value of a pixel of another color corresponding to the saturated pixel for calculation of an average color value.
 10. The image processing apparatus according to claim 1, wherein the combining unit is configured to combine only color components of the first image data and the second image data.
 11. An image processing method performed by an image processing apparatus, the image processing method comprising: determining a first white balance correction value corresponding to a first light source in captured image data and a second white balance correction value corresponding to a second light source in the image data; generating first image data by correcting the image data based on the first white balance correction value, and second image data by correcting the image data based on the second white balance correction value; calculating a color evaluation value of the image data; selecting one color evaluation region for determining a combination ratio from a plurality of color evaluation regions; comparing the color evaluation value of the image data with the selected color evaluation region under the second light source, and determining a combination ratio of the first image data and the second image data based on a result of the comparison; and combining the first image data and the second image data according to the determined combination ratio.
 12. A non-transitory computer-readable storage medium storing a program that causes a computer to execute a method comprising: determining a first white balance correction value corresponding to a first light source in captured image data and a second white balance correction value corresponding to a second light source in the image data; generating first image data by correcting the image data based on the first white balance correction value, and second image data by correcting the image data based on the second white balance correction value; calculating a color evaluation value of the image data; selecting one color evaluation region for determining a combination ratio from a plurality of color evaluation regions; comparing the color evaluation value of the image data with the selected color evaluation region under the second light source, and determining a combination ratio of the first image data and the second image data based on a result of the comparison; and combining the first image data and the second image data according to the determined combination ratio. 